Information processing apparatus and information processing system having a marker detecting unit and an extracting unit, and information processing method by using the same

ABSTRACT

Provided is an information processing apparatus, including: an image-taking unit configured to take an image of real scenery to thereby obtain real image; a marker-detecting unit configured to extract a marker image from the real image, the marker image being an image of marker projection light, the marker projection light being projected by a projection device to the real scenery in order to provide spatial first information, the first information being necessary to display virtual information such that the virtual information is superimposed on the real scenery, second information being added to the marker projection light; and an extracting unit configured to extract the second information added to the extracted marker image.

CROSS REFERENCES TO RELATED APPLICATIONS

The present application claims priority to Japanese Priority PatentApplication JP 2012-087184 filed in the Japan Patent Office on Apr. 6,2012, the entire content of which is hereby incorporated by reference.

BACKGROUND

The present disclosure relates to an information processing apparatus,an information processing method, and an information processing systemconfigured to display an image such that the image is superimposed onreal scenery.

There is known a technology called augmented reality (AR). According tothe AR technology, an image appropriate to real scenery is added to animage of the real scenery. According to the AR technology, a camera orthe like obtains an image of real scenery. Virtual information isdisplayed such that the virtual information is superimposed on theobtained image of real scenery. A user watches the virtual information,which is displayed such that the virtual information is superimposed onthe real scenery. As a result, the user recognizes as if an objectdisplayed as virtual information exists in the real scenery.

The AR technology includes a marker-type AR technology and amarkerless-type AR technology.

According to the marker-type AR technology, image information of amarker (for example, colored square having predetermined size) ispreviously registered. The marker is physically installed in realscenery. An image of real scenery is taken whereby an image of the realscenery (real image) is obtained. A marker is detected from the realimage. Spatial positional relation between the marker and animage-taking device is calculated based on information such as the size,angle, and the like of the detected marker. Based on the spatialpositional relation between the marker and the image-taking device, thedisplay position and the display angle of virtual information arecalculated. The virtual information is displayed on a display devicebased on the calculated display position and display angle. The relativeposition between the display device and the image-taking device isfixed. A user is capable of visually recognizing the virtual informationwith real scenery (for example, see Japanese Patent ApplicationLaid-open No. 2007-75213.).

Meanwhile, according to the markerless-type AR technology, a particularmarker is not used. An object or real scenery itself in real image isspatially recognized. Based on information such as the size, angle, andthe like of the object, the spatial positional relation between theobject and an image-taking device is calculated. Based on the positionalrelation, the display position and the display angle of the virtualinformation are calculated. The virtual information is displayed on adisplay device. The relative position between the display device and theimage-taking device is fixed. A user is capable of visually recognizingthe virtual information with real scenery.

SUMMARY

According to the marker-type AR technology, the display position and thedisplay angle of virtual information are calculated based on a marker,which exists in real scenery. Because of this, the marker-type ARtechnology has a merit in which it is possible to calculate the displayposition and the display angle of virtual information relatively easily.Meanwhile, the marker-type AR technology has the following demerits. Forexample, it requires time and effort to create markers. It is necessaryto secure marker-installation space. The quality of markers is declinedas time passes. Physical and psychological stress is caused by markersexisting in real scenery. There are limitations on design of markers.

To the contrary, the markerless-type AR technology has the followingmerits. That is, it is not necessary to create and install markers. Itis possible to apply the markerless-type AR technology to a location inwhich markers should not be installed. Meanwhile, the markerless-type ARtechnology has the following demerits. For example, complicatedcalculation is executed to construct a wide space model in the vicinityof displayed virtual information. High arithmetic capacity is required,and, if arithmetic capacity is inadequate, it is difficult to ensurestability and high accuracy, and calculation may be delayed.

Further, both the marker-type AR technology and the markerless-type ARtechnology have the following problems.

1. It is difficult for a user to control (move to different position,zoom, rotation, and the like) virtual information at will. That is, itis necessary for a user to stop to use an apparatus once, to move theposition of an existing marker (in case of marker-type AR technology),and to change the display position of virtual information by means of aprogram.

2. A visible-light camera recognizes an image. Because of this, it isnot possible to recognize an image in an extremely blight place and inan extremely dark place. In addition, an object shields a light source(sun, electric light, etc.), and a dark shadow (contrast) is thusgenerated on the surface of an existing object, which is alsoproblematic.

As described above, the marker-type AR technology and themarkerless-type AR technology have both merits and demerits. There isroom for improvement in preparation for practical use.

In view of the above-mentioned circumstances, it is desirable to displayvirtual information stably and with a high degree of accuracy.

According to an embodiment of the present application, there is providedan information processing apparatus, including: an image-taking unitconfigured to take an image of real scenery to thereby obtain realimage; a marker-detecting unit configured to extract a marker image fromthe real image, the marker image being an image of marker projectionlight, the marker projection light being projected by a projectiondevice to the real scenery in order to provide spatial firstinformation, the first information being necessary to display virtualinformation such that the virtual information is superimposed on thereal scenery, second information being added to the marker projectionlight; and an extracting unit configured to extract the secondinformation added to the extracted marker image.

A measurable characteristic may be added to the marker projection lightas the second information, and the extracting unit may be configured tomeasure the characteristic based on an image of the marker projectionlight, and to extract the second information.

The measurable characteristic of the marker projection light may be atleast one of light intensity (amplitude), wavelength (frequency), andblinking periods.

The information processing apparatus may further include an imagegenerating unit configured to generate, based on the first information,an image of virtual information displayed such that the virtualinformation is superimposed on the real scenery.

The second information may be identification information uniquelyidentifying the projection device, the extracting unit may be configuredto determine if the extracted identification information indicates thatthe projection device projects a display target of virtual information,and the image generating unit may generate, if the extracting unitdetermines that the extracted identification information indicates thatthe projection device projects a display target of virtual information,an image of the virtual information.

The information processing apparatus may further include a filtercapable of detecting a characteristic of the marker projection light.The marker-detecting unit may be configured to detect a marker imagefrom the filtered real image.

The information processing apparatus may further include a display unitconfigured to display an image of the virtual information such that theimage of the virtual information is superimposed on the real scenery.

The second information may be configuration changing information ofvirtual information displayed such that the virtual information issuperimposed on the real scenery, and the image generating unit may beconfigured to generate an image of the virtual information based on theconfiguration changing information of virtual information extracted bythe extracting unit.

The information processing apparatus may further include a sending unitconfigured to send the second information to the projection device, thesecond information being added to a marker image by the projectiondevice. The second information added to a marker image by the projectiondevice may be obtained by modulating the second information into themarker projection light to thereby add the second information to themarker image, the second information being received by the projectiondevice from the information processing apparatus, and the extractingunit may be configured to demodulate the second information added to themarker image.

The second information may be location information, the locationinformation specifying a location, object data to be displayed asvirtual information being stored in the location, and the informationprocessing apparatus may further include an object data obtaining unit,the object data obtaining unit being configured to obtain, based on thelocation information extracted by the extracting unit, object data to bedisplayed as virtual information.

According to an embodiment of the present application, there is providedan information processing method, including: taking, by an image-takingunit, an image of real scenery to thereby obtain real image; extracting,by a marker-detecting unit, a marker image from the real image, themarker image being an image of marker projection light, the markerprojection light being projected by a projection device to the realscenery in order to provide spatial first information, the firstinformation being necessary to display virtual information such that thevirtual information is superimposed on the real scenery, secondinformation being added to the marker projection light; and extracting,by an extracting unit, the second information added to the extractedmarker image.

According to an embodiment of the present application, there is providedan information processing system, including: a projection device capableof projecting a marker to real scenery, second information being addedto the marker; and an information processing apparatus including animage-taking unit configured to take an image of the real scenery tothereby obtain real image, a marker-detecting unit configured to extracta marker image from the real image, the marker image being an image ofmarker projection light, the marker projection light being projected bya projection device to the real scenery in order to provide spatialfirst information, the first information being necessary to displayvirtual information such that the virtual information is superimposed onthe real scenery, second information being added to the markerprojection light, and an extracting unit configured to extract thesecond information added to the extracted marker image.

The information processing apparatus may further include a sending unitconfigured to send the second information to the projection device, thesecond information being added to a marker image by the projectiondevice, the projection device may include a receiving unit configured toreceive the second information from the information processingapparatus, and a modulation unit configured to modulate the secondinformation received by the receiving unit into the marker projectionlight to thereby add the second information to a marker image, and theextracting unit may be configured to demodulate the second informationadded to the marker image.

As described above, according to the present application, it is possibleto display virtual information stably and with a high degree ofaccuracy.

Additional features and advantages are described herein, and will beapparent from the following Detailed Description and the figures.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a schematic diagram showing an information processing systemaccording to a first embodiment of the present application;

FIG. 2 is a block diagram showing the hardware configuration of an HMDand the hardware configuration of an input device;

FIG. 3 is a block diagram showing the functional configuration of theHMD for executing first process;

FIG. 4 is a diagram schematically showing the principle of generating asuperimposition parameter by a superimposition-parameter generatingunit;

FIG. 5 is a flowchart showing behaviors of the first process executed bythe HMD;

FIG. 6 is a diagram schematically showing the first process;

FIG. 7 is a diagram schematically showing the third process;

FIG. 8 is a block diagram showing the functional configuration of theHMD for executing the third process;

FIG. 9 is a block diagram showing the hardware configuration of theinput device for executing the third process;

FIG. 10 is a schematic diagram showing an information processing systemaccording to a modified example 2;

FIG. 11 is a block diagram showing the hardware configuration of aninformation processing system according to the modified example 2;

FIG. 12 is a schematic diagram showing an information processing systemaccording to a modified example 3;

FIG. 13 is a schematic diagram showing an information processing systemaccording to a modified example 4;

FIG. 14 is a block diagram showing the hardware configuration of theinformation processing system according to the modified example 4;

FIG. 15 is a schematic diagram showing an information processing systemaccording to a modified example 5; and

FIG. 16 is a schematic diagram showing an information processing systemaccording to a modified example 6.

DETAILED DESCRIPTION

Hereinafter, an embodiment of the present disclosure will be describedwith reference to the drawings.

<First Embodiment>

[Gist of First Embodiment]

FIG. 1 is a schematic diagram showing an information processing system 1according to a first embodiment of the present application.

The information processing system 1 of this embodiment includes at leastone Head Mount Display (HMD) 100 (information processing apparatus) andat least one input device 200 (projection device).

The shape of the HMD 100 is similar to glasses as a whole. A user Uwears the HMD 100 on his head. The HMD 100 includes a display unit 102and an image-taking unit 101. The display unit 102 is arranged in frontof the eyes of the user U when the user U wears the HMD 100. Theimage-taking unit 101 is capable of taking an image of at least a fieldof vision of the user U in real scenery. The display unit 102 haspermeability. The display unit 102 allows the user U himself to visuallyrecognize real scenery through the display unit 102. At the same time,the display unit 102 is capable of displaying an image such that theimage is superimposed on the real scenery, which the user U visuallyrecognizes. The HMD 100 detects a marker M in an image of real scenery(real image) taken by the image-taking unit 101. The marker M isprojected on the real scenery by the input device 200. The input device200 projects the marker M to provide spatial information (firstinformation) necessary to display virtual information such that thevirtual information is superimposed on the real scenery. The HMD 100calculates a superimposition parameter of virtual information I based onthe detected marker M. The “superimposition parameter” is a parameter ofa configuration of virtual information displayed such that the virtualinformation is superimposed on real scenery. Specifically, the“superimposition parameter” includes the position, size, and angle ofthe virtual information I. The HMD 100 generates predetermined virtualinformation based on the calculated superimposition parameter. Thedisplay unit 102 displays the virtual information I such that thevirtual information I is superimposed on the marker M, which isprojected on real scenery visually recognized by the user U. The userpreviously inputs selection of content to be displayed as the virtualinformation I by means of an input unit (FIG. 2) of the HMD 100.

The input device 200 has a size and a shape such that the user U graspsthe input device 200 with his hand. The input device 200 includes aprojection button 201, zoom sliders 202, and a power button 203. Whenthe user U presses the projection button 201, a projection window 204projects a graphic having a predetermined shape on a projection targetobject T (desk, etc.) in real scenery. The graphic is the marker M,which provides spatial information (first information) necessary todisplay virtual information such that the virtual information issuperimposed on the real scenery. The projection position of the markerM is the display position of the virtual information I displayed by theHMD 100. In addition, the user U moves the input device 200 whileholding down the projection button 201. As a result, the user U iscapable of controlling the virtual information I. For example, the userU moves the input device 200 while holding down the projection button201 to thereby move the projection position of the marker M. As aresult, the virtual information I moves (is dragged). Similarly, theuser U rotates the input device 200 while holding down the projectionbutton 201 to thereby rotate the marker M. As a result, the virtualinformation I rotates. Further, the user U operates the zoom sliders 202while holding down the projection button 201. As a result, the virtualinformation I is enlarged/reduced (zoomed). The user U wishes to displayother new virtual information while the virtual information I is beingdisplayed. In this case, the user U stops pressing the projection button201. Then, the user U inputs selection of content to be displayed as thenew virtual information by using the input unit (FIG. 2) of the HMD 100.Further, the user U stops pressing the projection button 201 and thenpresses the projection button 201 again, to thereby superimpose themarker M on the displayed virtual information I. Then, the user U iscapable of operating the virtual information I again.

[Hardware Configuration of HMD]

FIG. 2 is a block diagram showing the hardware configuration of the HMD100 and the hardware configuration of the input device 200.

The HMD 100 includes a CPU (Central Processing Unit) 103 and a built-inpower source 110. The HMD 100 further includes a memory 104, theimage-taking unit 101, the display unit 102, an input unit 105, a motionsensor 106, an environment sensor 107, a first sending/receiving device108, and a second sending/receiving device 109, each of which isconnected to the CPU 103.

The CPU 103 executes various processes in accordance with programsstored in the memory 104.

The image-taking unit 101 is capable of taking an image of at least auser's field of vision in real scenery. The image-taking unit 101includes an image sensor such as a CMOS (Complementary Metal OxideSemiconductor) image sensor, and an A/D converter configured to A/D(Analog/Digital) convert an output from the image sensor.

The display unit 102 includes an LCD (Liquid Crystal Display device), anoptical system, and the like. The display unit 102 shows an image formedby the LCD to a user via the optical system. More specifically, thedisplay unit 102 is capable of displaying an image formed by the LCDsuch that the image is superimposed on a user's field of vision whileallowing the user to visually recognize the outwards.

The input unit 105 includes, for example, a button, a slider, a switch,a dial, a touchscreen, and the like. The input unit 105 is capable ofreceiving instructions to the CPU 103 and selection of content to bedisplayed as virtual information, which are input through user'soperations.

The motion sensor 106 is, for example, an acceleration sensor, a gyrosensor, or a magnetic sensor. The motion sensor 106 is capable ofdetecting movement of the HMD 100.

The environment sensor 107 is capable of detecting illuminance,temperature, and humidity, for example.

The first sending/receiving device 108 is a medium/high-speedclose-range wireless sending/receiving device such as, for example,Bluetooth (registered trademark) or Wi-Fi (registered trademark). Thefirst sending/receiving device 108 sends/receives information to/fromthe input device 200.

The second sending/receiving device 109 is a medium-range wirelesssending/receiving device such as, for example, 3G (3rd Generation) orWiMAX (Worldwide Interoperability for Microwave Access, registeredtrademark).

The second sending/receiving device 109 is connected to a network N suchas the Internet or a LAN (Local Area Network). The secondsending/receiving device 109 accesses a content server connected to thenetwork N, and, for example, downloads content to be displayed asvirtual information.

[Hardware Configuration of Input Device]

With reference to FIG. 2, the input device 200 includes a CPU 212 and abuilt-in power source 211. The input device 200 further includes amemory 205, an input unit 206, a motion sensor 207, a thirdsending/receiving device 208, a modulation unit 209, and a projectionunit 210, each of which is connected to the CPU 212.

The CPU 212 executes various processes in accordance with programsstored in the memory 205. Note that an MPU (Micro Processing Unit) maybe used instead of a CPU.

The input unit 206 includes buttons (such as the projection button 201,the zoom sliders 202, and the power button 203), a slider, a switch, adial, a touchscreen, and the like. The input unit 206 is capable ofreceiving instructions to the CPU 212, which are input through user'soperations.

The motion sensor 207 is, for example, an acceleration sensor, a gyrosensor, or a magnetic sensor. The motion sensor 207 is capable ofdetecting movement of the input device 200.

The third sending/receiving device 208 is a medium/high-speedclose-range wireless sending/receiving device such as, for example,Bluetooth (registered trademark) or Wi-Fi (registered trademark). Thethird sending/receiving device 208 sends/receives information to/fromthe HMD 100.

The modulation unit 209 modulates digital data treated by the CPU 212into optical signals that the projection unit 210 is capable ofprojecting.

The projection unit 210 is a laser pointer including a laser lightsource and an optical system. The optical system is capable ofprojecting, from the projection window 204 (FIG. 1), laser light emittedby the laser light source on a projection target object (wall, desk,etc.) in real scenery as a graphic having a predetermined shape being amarker. As the laser pointer, a point-display-type laser pointer (red,green, blue), a beam-variable-type (scanner-type, lens-type,hologram-type) laser pointer, an ultraviolet-type laser pointer, aninfrared-type laser pointer, or the like may be employed.

[Processes Executed by HMD]

The HMD 100 and the input device 200 are capable of executing thefollowing three processes.

1. The input device 200 adds information (second information) specifyingthe input device 200 itself to a marker. The HMD 100 specifies the inputdevice 200, which projects a marker in real image data, based on animage of the marker taken by the HMD 100. In a case where the marker isprojected by the particular input device 200, the HMD 100 displaysvirtual information (first process).

2. The input device 200 adds, to a marker, configuration changinginformation (second information) of virtual information displayed suchthat the virtual information is superimposed on the real scenery. TheHMD 100 specifies the configuration changing information of virtualinformation based on an image of the marker taken by the HMD 100. TheHMD 100 changes the configuration of virtual information based on thespecified changing information (second process). Specifically, theconfiguration of virtual information is display position, size, and thelike of displayed virtual information.

3. The input device 200 adds, to a marker, location information (secondinformation). The location information (second information) specifies alocation on a network in which object data is stored. The HMD 100specifies a location on a network of object data based on an image ofthe marker taken by the HMD 100, downloads virtual information from thenetwork, and displays the virtual information (third process).

<First Process>

[Gist of First Process]

To solve the above-mentioned problems of the marker-type AR technologyand the markerless-type AR technology, there is disclosed a method ofprojecting a marker from a handheld laser pointer held by a user(Japanese Patent Application Laid-open No. 2007-272623). Here, let'sassume that a plurality of users display virtual information on the sameprojection target object (desk, wall, etc.), and that the plurality ofusers control the virtual information simultaneously by using inputdevices (laser pointers), respectively. In this case, the followingprocesses and the like are executed. Each of the users projects apointing point on the projection target object by using the inputdevice. The user moves the input device to thereby move the pointingpoint projected on the projection target object. The virtual informationdisplayed on the pointing point moves as the pointing point moves. Inthis case, let's assume that a plurality of users use input devices,which project pointing points having the same shape. In this case, animage-taking device is not capable of determining which pointing pointis projected by an input device of which user. As a result, for example,based on a pointing point projected by an input device of an irrelevantuser, virtual information may be displayed to another user.

Meanwhile, according to Japanese Patent Application Laid-open No.H07-200160, a plurality of users control virtual information by usinglaser pointers as input devices, respectively. In this situation, theinput devices change project patterns of the pointing points,respectively. As a result, it is possible to distinguish between theinput devices of the plurality of users. However, according to themethod, it is necessary to physically change the shapes of the projectpatterns of the input devices. This method has limitations from theviewpoint of technology and cost, which is problematic. Specifically,this is a large problem in a case where the number of input devices usedat the same time is large, in a case of a configuration capable ofchanging the project pattern during input operation, and the like.

Here, let's discuss a case where a laser projector is used as an inputdevice instead of a laser pointer. A laser projector includes, forexample, a built-in MEMS (Micro Electro Mechanical System) scanner. Thelaser projector scans from top to bottom by means of one laser light,which moves from side to side at high speed. The laser projector iscapable of projecting a graphic having an arbitrary shape and anarbitrary image. Meanwhile, a laser projector requires hardware/softwareconfigurations more complex than those of a laser pointer, whichprojects a simple graphic, which is problematic. As a result, a laserprojector itself may be large, costs may rise, and power consumption maybe increased, which are problematic. Further, even in a case where theshapes of graphics projected by a plurality of input devices aredifferent from each other, if projected graphics are overlapped eachother, it is difficult to discriminate the projected graphics, which isproblematic.

In view of the above-mentioned circumstances, in the first process, theHMD determines which input device projects a marker, whose image istaken by the HMD, stably and with a high degree of accuracy. As aresult, the HMD determines a marker, on which virtual information is tobe displayed, stably and with a high degree of accuracy. As a result, itis possible to display virtual information stably and with a high degreeof accuracy.

[Input Device]

The input device 200 is configured to project a marker, to whichinformation (identification information) uniquely identifying the inputdevice itself is added. The identification information may be, forexample, a serial number or the like set when manufacturing the inputdevice 200, or a user name or the like set by a user by using the inputunit 206. More specifically, the input device 200 is configured toproject a marker having at least one of unique light intensity(amplitude), unique wavelength (frequency), and unique blinking periods.It is possible to uniquely identify the input device 200 based onindividual differences of light intensity (amplitude), wavelength(frequency), and blinking periods.

[Functional Configuration of HMD for Executing First Process]

FIG. 3 is a block diagram showing the functional configuration of theHMD 100 for executing the first process.

The HMD 100 includes the image-taking unit 101, a marker-detecting unit121, an additional-information obtaining unit 125 (extracting unit), asuperimposition-parameter generating unit 122, a transformation-matrixcalculating unit 123, an image-data generating unit 124, and the displayunit 102.

The image-taking unit 101 takes an image of real scenery, and obtainsreal image data. The image-taking unit 101 supplies the obtained realimage data to the marker-detecting unit 121 and theadditional-information obtaining unit 125.

The additional-information obtaining unit 125 obtains the real imagedata from the image-taking unit 101. The additional-informationobtaining unit 125 processes the image to thereby measurecharacteristics of the marker. Specifically, the characteristics of themarker include at least one of light intensity (amplitude), wavelength(frequency), and blinking periods. The additional-information obtainingunit 125 determines identification information of the input device 200,which is assigned to the measured characteristics. Theadditional-information obtaining unit 125 determines if the input device200, which is represented by the determined identification information,is a projection target of virtual information. Here, theadditional-information obtaining unit 125 previously registers thatwhich input device 200 projects a marker on which virtual information isdisplayed. In other words, at least one input device 200 as a projectiontarget of virtual information is previously registered in theadditional-information obtaining unit 125. For example, a user sets aprojection target of virtual information by using the input unit 105.The additional-information obtaining unit 125 notifies thesuperimposition-parameter generating unit 122 of the determinationresult.

The marker-detecting unit 121 detects a marker projected by the inputdevice 200 from the real image data obtained by the image-taking unit101. Information on a reference marker is previously registered in themarker-detecting unit 121. The “reference marker” is a marker having apredetermined reference shape in a case where the marker is projected ata predetermined distance in a perpendicular direction. The “informationon a reference marker” includes the size of the reference marker,distances between respective corners, lengths of respective sides, andthe like. The marker-detecting unit 121 generates a plane-coordinatetransformation matrix based on the size of the reference marker.According to the plane-coordinate transformation matrix, a markerdetected from real image data coincides with the shape of a referencemarker. The marker-detecting unit 121 executes coordinate transformationwith respect to the detected marker by using the plane-coordinatetransformation matrix. Subsequently, the marker-detecting unit 121executes pattern matching between the coordinate-transformed marker andthe reference marker. The marker-detecting unit 121 determines a degreeof coincidence between the detected marker and the reference marker. Themarker-detecting unit 121 supplies the determination result to thesuperimposition-parameter generating unit 122.

The superimposition-parameter generating unit 122 calculates spatialpositional relation (i.e., angle and distance) between a marker and theimage-taking unit 101 based on distortion of the marker with respect tothe reference marker. Here, the marker is projected by the input device200, which is a projection target of virtual information. The marker hasa predetermined degree of coincidence. The marker is projected on aprojection target object (wall, desk, etc.). Further, as shown in FIG.4, the superimposition-parameter generating unit 122 calculates acoordinate system (A) of the marker M based on an outline extractionmethod. The superimposition-parameter generating unit 122 calculates asuperimposition parameter of virtual information such that a coordinatesystem (C) is attained, based on the spatial positional relation betweenthe marker and the image-taking unit 101. In the coordinate system (C),the coordinate system (A) of the marker M coincides with apreviously-set coordinate system (B) of the virtual information I. Thesuperimposition-parameter generating unit 122 corrects thesuperimposition parameter such that virtual information is displayedmore naturally to a user, based on the position of the HMD 100 detectedby the motion sensor 106 and based on the position of the input device200 detected by the motion sensor 106 of the input device 200. Further,the superimposition-parameter generating unit 122 corrects thesuperimposition parameter based on positional relation between the eyesof the user and the display unit 102.

The transformation-matrix calculating unit 123 generates aspace-coordinate transformation matrix. The space-coordinatetransformation matrix is used to convert a coordinate system in which amarker is a reference into a coordinate system in which the image-takingunit 101 in real scenery is a reference, based on the superimpositionparameter. The image-data generating unit 124 executes coordinatetransformation of object data of previously-recorded virtual informationby using the space-coordinate transformation matrix supplied from thetransformation-matrix calculating unit 123. As a result, the image-datagenerating unit 124 calculates (draws) object image data of virtualinformation in a coordinate system in which the image-taking unit 101 isa reference.

The image-data generating unit 124 supplies the generated object imagedata of virtual information into the display unit 102.

The display unit 102 displays the object image data of the virtualinformation supplied from the image-data generating unit 124.

[Behaviors of First Process Executed by HMD]

FIG. 5 is a flowchart showing behaviors of the first process executed bythe HMD 100.

The CPU 103 executes a predetermined initializing process (Step S101).Then, the image-taking unit 101 takes an image of real scenery, andobtains real image data (Step S102). The image-taking unit 101 suppliesthe obtained real image data to the marker-detecting unit 121 and theadditional-information obtaining unit 125.

The marker-detecting unit 121 detects a marker projected by the inputdevice 200 from the real image data obtained (Step S102) by theimage-taking unit 101 (Step S103). The marker-detecting unit 121generates a plane-coordinate transformation matrix based on the size ofthe reference marker. According to the plane-coordinate transformationmatrix, a marker detected from real image data coincides with the shapeof a reference marker. The marker-detecting unit 121 executes coordinatetransformation with respect to the detected marker by using theplane-coordinate transformation matrix. Subsequently, themarker-detecting unit 121 executes pattern matching between thecoordinate-transformed marker and the reference marker. Themarker-detecting unit 121 determines a degree of coincidence between thedetected marker and the reference marker (Step S104). Themarker-detecting unit 121 supplies the determination result to thesuperimposition-parameter generating unit 122.

Meanwhile, the additional-information obtaining unit 125 also obtainsthe real image data from the image-taking unit 101. Theadditional-information obtaining unit 125 processes the image to therebymeasure characteristics of a marker, i.e., at least one of lightintensity (amplitude), wavelength (frequency), and blinking periods. Theadditional-information obtaining unit 125 determines identificationinformation of the input device 200 (Step S105). The identificationinformation is assigned to the measured characteristics. Theadditional-information obtaining unit 125 determines if the input device200, which is indicated by the determined identification information,coincides with a projection target of virtual information (Step S 106).The additional-information obtaining unit 125 notifies thesuperimposition-parameter generating unit 122 of the determinationresult.

The superimposition-parameter generating unit 122 estimates spatialpositional relation between a marker and the image-taking unit 101 basedon distortion of the marker with respect to the reference marker. Here,the marker is projected by the input device 200, which is a projectiontarget of virtual information (Step S106, Yes), and has a predetermineddegree of coincidence (Step S104, Yes). Specifically, thesuperimposition-parameter generating unit 122 calculates spatialpositional relation between the image-taking unit 101 and a markerprojected on a projection target object (wall, etc.), i.e., angle anddistance. Further, the superimposition-parameter generating unit 122calculates a superimposition parameter of the virtual information suchthat the coordinate system of the marker coincides with the coordinatesystem of the virtual information (Step S107). Then, thesuperimposition-parameter generating unit 122 corrects thesuperimposition parameter such that the virtual information is displayedmore naturally to a user (Step S108).

The transformation-matrix calculating unit 123 generates aspace-coordinate transformation matrix (Step S 109). Thespace-coordinate transformation matrix is used to convert a coordinatesystem in which a marker is a reference into a coordinate system inwhich the image-taking unit 101 in real scenery is a reference, based onthe superimposition parameter. The image-data generating unit 124executes coordinate transformation of object data of previously-recordedvirtual information by using the space-coordinate transformation matrixsupplied from the transformation-matrix calculating unit 123. As aresult, the image-data generating unit 124 calculates (draws) objectimage data of virtual information in a coordinate system in which theimage-taking unit 101 is a reference (Step S110). The image-datagenerating unit 124 supplies the generated object image data of virtualinformation into the display unit 102. The display unit 102 displays thesupplied object image data of the virtual information (Step S111). Afterthat, processes from obtainment of image data of real scenery (StepS102) to display of object image data of virtual information (Step S111)are repeated with respect to the subsequent frames.

[Effects of First Process]

As described above, according to the first process, the followingeffects may be obtained.

1. If an HMD is not capable of identifying which input device emitswhich marker, the HMD may display virtual information with respect to amarker projected by an input device of an irrelevant user, for example

To the contrary, according to the first process, as shown in FIG. 6,HMDs 100A, 100B are capable of identifying identification information ofan input device 200A, which is added to a marker ML. Each of the HMDs100A, 100B is capable of displaying virtual information Ia on a markerM. However, a HMD 100C is not capable of identifying identificationinformation of the input device 200A, which is added to the marker ML.The HMD 100C is not capable of displaying virtual information on themarker M, even if obtained real image data includes the marker M. Inthis manner, it is possible to allow the HMD 100A and the HMD 100B todisplay virtual information. It is possible not to allow the HMD 100C todisplay virtual information. The HMD 100A is owned by a user U1 whoholds an input device 200A, which projects the marker M. The HMD 100B isowned by a user U2 who relates to the user U1. The HMD 100C is owned byan irrelevant user U3.

2. According to a technology in which the shapes of markers aredifferent from each other and each input device is distinguished, in acase where projected markers are overlapped with each other on aprojection target object, it is difficult to identify each marker. As aresult, it is difficult to identify each input device, which projectseach marker, which is problematic.

To the contrary, according to the first process, identificationinformation identifying an input device is added to a marker. So it ispossible to determine an input device, which projects a marker, stablyand with a high degree of accuracy based on at least one of lightintensity (amplitude), wavelength (frequency), and blinking periods ofthe marker irrespective of overlapped markers. Further, the firstprocess has the following merits. That is, this embodiment is lower incost than the technology in which the shapes of markers emitted frominput devices are different from each other physically. This embodimentcan be realized by employing hardware/software configurations simplerthan those of a technology, which uses laser pointers capable ofprojecting markers having various shapes.

<Second Process>

Hereinafter, configurations and the like different from those alreadydescribed will be mainly described. Repetition in descriptions will beavoided.

In the first process, identification information identifying the inputdevice 200 is added as additional information added to a marker. To thecontrary, in the second process, configuration changing information isadded as additional information added to a marker. The configurationchanging information is of virtual information to be displayed such thatthe virtual information is superimposed on real scenery.

[Input Device]

A user inputs a predetermined operation in the input unit 206 of theinput device 200 (e.g., presses button). Then, the CPU 212 determinesthat configuration changing information of marker display (e.g., tomove, rotate, enlarge/reduce (zoom) marker, etc.) is input. Here,characteristics (at least one of light intensity (amplitude), wavelength(frequency), and blinking periods) of a marker are uniquely assigned tothe configuration changing information of marker display (e.g., to move,rotate, enlarge/reduce (zoom) marker, etc.). For example, increase offrequency is assigned to enlarging a marker, and decrease of frequencyis assigned to reducing a marker. The CPU 212 determines thatconfiguration changing information of marker display is input in theinput unit 206. Then, the CPU 212 changes a projected marker based oncharacteristics (at least one of light intensity (amplitude), wavelength(frequency), and blinking periods) assigned to the configurationchanging information of marker display.

[HMD]

The additional-information obtaining unit 125 of the HMD 100 processesan image to thereby measure characteristics (at least one of lightintensity (amplitude), wavelength (frequency), and blinking periods) ofa marker in real image data obtained from the image-taking unit 101. Theadditional-information obtaining unit 125 detects that thecharacteristics are changed. Then, the additional-information obtainingunit 125 determines configuration changing information of marker displayassigned to the changed characteristics. Here, how to change virtualinformation displayed on a marker according to the way of changingcharacteristics (at least one of light intensity (amplitude), wavelength(frequency), and blinking periods) of the marker is previouslyregistered in the additional-information obtaining unit 125. Forexample, enlarging a marker is registered for increase of frequency, andreducing a marker is registered for decrease of frequency. Theadditional-information obtaining unit 125 notifies thesuperimposition-parameter generating unit 122 of the determinationresult.

Note that instructions (enter, return, etc.) to an application may beadded as additional information to a marker emitted from the inputdevice 200, in addition to configuration changing information of virtualinformation. The HMD 100 may execute an application based on aninstruction to the application added to a marker.

[Effect of Second Process]

As described above, according to the second process, the followingeffect may be obtained.

Even if a user does not hold the input device 200 with his hand and doesnot move the input device 200, it is possible to input configurationchanging information (e.g., to move or rotate virtual information, etc.)of virtual information only by inputting a predetermined operation inthe input device 200. Because of this, it is possible to inputconfiguration changing information of virtual information in a statewhere the input device 200 is on a table or the like, for example. Theuser-friendliness is thus improved.

<Third Process>

FIG. 7 is a diagram schematically showing the third process.

A user U1 has an input device 200A, which projects the marker M, andwears an HMD (hereinafter, referred to as “projection HMD”) 100A. Theprojection HMD 100A supplies location information to the input device200A. The location information specifies a location on a network, inwhich object data to be displayed as virtual information is stored. Theinput device 200A adds the obtained location information to a marker.Users U2, U3 do not have the input device 200A, which projects themarker M, and wear HMDs (hereinafter, referred to as “non-projectionHMDs”) 100B, 100C. Each of the non-projection HMDs 100B, 100C extractslocation information from a marker in a taken image. Each of thenon-projection HMDs 100B, 100C is capable of downloading object data tobe displayed as virtual information from a content server 300 connectedto the network N based on the location information.

[Functional Configuration of HMD for Executing Third Process]

FIG. 8 is a block diagram showing the functional configuration of theHMD 100 for executing the third process.

The HMD 100 includes a location information obtaining unit 130, thefirst sending/receiving device 108, the image-taking unit 101, ademodulator unit 132, an object data obtaining unit 131, and the secondsending/receiving device 109.

The location information obtaining unit 130 and the firstsending/receiving device 108 execute (1) functions specific to theprojection HMD. The image-taking unit 101 and the demodulator unit 132execute (2) functions specific to the non-projection HMD 100. The objectdata obtaining unit 131 and the second sending/receiving device 109execute (3) functions common to the projection HMD 100 and thenon-projection HMD 100.

[(1) Functions Specific to Projection HMD]

The location information obtaining unit 130 obtains locationinformation. The location information specifies a location in thenetwork N, in which object data to be displayed as virtual informationis stored. The “location information” is, specifically, URL (UniformResource Locator), local network path, or the like. For example, thelocation information obtaining unit 130 may obtain location informationvia the network N by using the second sending/receiving device 109.Alternatively, the location information obtaining unit 130 may obtainlocation information input by a user by using the input unit 105. Thelocation information obtaining unit 130 supplies the obtained locationinformation to the object data obtaining unit 131. Further, the locationinformation obtaining unit 130 encodes the obtained location informationto thereby generate encoded information. The location informationobtaining unit 130 sends the generated encoded information to the inputdevice 200 by using the first sending/receiving device 108.

[(2) Functions Specific to Non-Projection HMD 100]

The image-taking unit 101 takes an image of real scenery to therebyobtain real image data. The image-taking unit 101 supplies the obtainedreal image data to the demodulator unit 132.

The demodulator unit 132 processes an image to thereby measure change oflight intensity (amplitude) or wavelength (frequency) of a marker inreal image data. The demodulator unit 132 extracts encoded informationbased on the measurement result. The demodulator unit 132 decodes theextracted encoded information to thereby obtain location information asadditional information. That is, the demodulator unit 132 functions asan additional-information obtaining unit. The demodulator unit 132supplies location information to the object data obtaining unit 131.

[(3) Functions Common to Projection HMD 100/Non-Projection HMD 100]

The object data obtaining unit 131 downloads object data from thecontent server 300 via the network N based on location informationobtained from the location information obtaining unit 130 or from thedemodulator unit 132 by using the second sending/receiving device 109.

[Hardware Configuration of Input Device for Executing Third Process]

FIG. 9 is a block diagram showing the hardware configuration of theinput device 200 for executing the third process.

The input device 200 includes the CPU 212, the input unit 206, the thirdsending/receiving device 208, the modulation unit 209, a laser lightsource 220, and an optical system 222. The laser light source 220 andthe optical system 222 are included in the projection unit 210.

The CPU 212 detects an operation input in the input unit 206 by a user,and turns on/off the laser light source 220. Further, the CPU 212obtains encoded information from the HMD 100 by using the thirdsending/receiving device 208. The encoded information is informationobtained by encoding location information. The CPU 212 generatesmodulation information. The modulation information is information usedto modulate (amplitude modulate or frequency modulate) the encodedinformation into a marker. The CPU 212 supplies the generated modulationinformation to the modulation unit 209.

The modulation unit 209 modulates (amplitude modulates or frequencymodulates) a marker emitted from the laser light source 220 based on themodulation information obtained from the CPU 212. As a result, encodedinformation is added. The encoded information is information obtained byencoding location information into a marker.

The optical system projects the marker obtained by modulation on realscenery by using the projection window 204.

Note that, according to this embodiment, the HMD 100 encodes locationinformation to thereby generate encoded information, and sends theencoded information to the input device 200. Alternatively, the HMD 100may send location information to the input device 200, and the inputdevice 200 may encode the location information to thereby generateencoded information.

Further, according to this embodiment, encoded information of locationinformation is additional information. The location informationspecifies a location in a network, in which object data to be displayedas virtual information is stored. Alternatively, encoded information ofidentification information of the input device 200 described in thefirst process or encoded information of configuration changinginformation of virtual information described in the second process maybe additional information.

[Effect of Third Process]

As described above, according to the third process, the following effectmay be obtained.

Let's assume that an HMD (referred to as “data holding HMD”) of a userhaving an input device, which projects a marker, holds object data ofvirtual information to be displayed on the marker, and that each of aplurality of HMDs (referred to as “no-data holding HMDs”) displays theobject. In this case, in a case where each of a plurality of no-dataholding HMDs obtains object data, for example, the following method orthe like may be employed. That is, each of a plurality of no-dataholding HMDs obtains data from the data holding HMD via wirelesscommunication or the like. However, according to this method, ifcommunication is not established (e.g., no-data holding HMD is notcapable of accessing data holding HMD), the no-data holding HMD is notcapable of displaying virtual information.

To the contrary, according to the third process, it is not necessary forHMDs to directly communicate with each other and to share object data.It is possible for each of an unspecified number of HMDs to extractlocation information from real image data, to obtain object data from anetwork based on the location information, and to display the sameobject. It is possible to apply such a technology to, for example, thepurpose of showing signage to an unspecified number of users in a publicarea, and other purposes.

MODIFIED EXAMPLE 1

The modified example described below is a case where amplitude(intensity) of light projected by the input device 200 in the aboveembodiment is changed to thereby add information. The HMD 100 mayfurther include a filter (blinking period synchronous PLL (Phase-LockedLoop), differential filter, or the like) capable of detecting lightintensity change pattern of a marker projected by the input device 200.In this case, the marker-detecting unit 121 detects a marker in thefiltered real image data.

The modified example described below is a case where frequency(wavelength) of light projected by the input device 200 in the aboveembodiment is changed to thereby add information. The HMD 100 mayfurther include a filter (wavelength selection filter, differentialfilter, or the like) capable of detecting wavelength change pattern of amarker projected by the input device 200. In this case, themarker-detecting unit 121 detects a marker in the filtered real imagedata.

In a case where a filter is not used, recognition rate (S/N(Signal/Noise)) of a marker in real image data recognized by the HMD 100is decreased affected by ambient light such as intense sun light(afternoon sunlight, metallic reflection light) or intense electriclight. As a result, recognition errors or calculation errors may occur.However, it is possible to execute display and input operations ofvirtual information stably and with a high degree of accuracy underambient light by using a filter.

MODIFIED EXAMPLE 2

FIG. 10 is a schematic diagram showing an information processing system3 according to the modified example 2. FIG. 11 is a block diagramshowing the hardware configuration of the information processing system3 according to the modified example 2.

The information processing system 3 includes an HMD 100 b, an inputdevice 200 a, and a mobile information terminal 400. In the aboveembodiment, the CPU 103 in the HMD 100, 100 a executes main process. Tothe contrary, according to the information processing system of thismodified example, the mobile information terminal 400, which isindependent of the HMD 100 b, executes main process. As the mobileinformation terminal 400, for example, a smartphone or a handheld gamemachine may be employed.

The hardware configuration of the HMD 100 b is similar to the hardwareconfiguration of the HMD 100 of the first embodiment except that the HMD100 b does not include the first sending/receiving device 108 and thesecond sending/receiving device 109 and further includes a fifthsending/receiving device 112. The fifth sending/receiving device 112 isa medium/low-speed close-range wireless sending/receiving device suchas, for example, Bluetooth (registered trademark). The fifthsending/receiving device 112 sends/receives information to/from themobile information terminal 400. More specifically, the fifthsending/receiving device 112, for example, sends an image input signalof a real image obtained by an image-taking unit to the mobileinformation terminal 400.

The hardware configuration of the input device 200 a is similar to thehardware configuration of the input device 200 of the first embodimentexcept that the input device 200 a does not include the thirdsending/receiving device 208 and further includes a sixthsending/receiving device 213. The sixth sending/receiving device 213 isa close-range wireless sending/receiving device such as Bluetooth(registered trademark) or infrared. The sixth sending/receiving device213 sends/receives information to/from the mobile information terminal400. More specifically, the sixth sending/receiving device 213, forexample, sends an operation (e.g., zoom operation) input signal input inan input unit by a user, to the mobile information terminal 400.

The HMD 100 b does not include the first sending/receiving device 108,and the input device 200 a does not include the second sending/receivingdevice. Because of this, the HMD 100 b do not send/receive informationto/from the input device 200 a directly, and vice versa. The HMD 100 bsends/receives information to/from the input device 200 a via the mobileinformation terminal 400, and vice versa.

The mobile information terminal 400 includes a CPU 401 and a built-inpower source 407. The mobile information terminal 400 further includes amemory 402, a display unit 403, an input unit 404, a seventhsending/receiving device 405, an eighth sending/receiving device 406,and a ninth sending/receiving device 408, each of which is connected tothe CPU 401.

The CPU 401 executes various processes as function units described inthe above embodiment in accordance with programs stored in the memory402.

The seventh sending/receiving device 405 is a medium/low-speedclose-range wireless sending/receiving device such as, for example,Bluetooth (registered trademark). The seventh sending/receiving device405 sends/receives information to/from the HMD 100 b. More specifically,the seventh sending/receiving device 405, for example, sends an imageoutput signal of virtual information to be displayed by a display unitof the HMD 100 b, to the HMD 100 b.

The eighth sending/receiving device 406 is a close-range wirelesssending/receiving device such as Bluetooth (registered trademark) orinfrared. The eighth sending/receiving device 406 sends/receivesinformation to/from the input device 200 a. More specifically, theeighth sending/receiving device 406, for example, sends a change signalto the input device 200 a. The change signal is used to change a patternof a graphic as a marker projected by the input device 200 a.

The ninth sending/receiving device 408 is a medium-range wirelesssending/receiving device such as, for example, 3G (3rd Generation) orWiMAX (Worldwide Interoperability for Microwave Access, registeredtrademark). The ninth sending/receiving device 408, for example,connects the network N such as the Internet or LAN (Local Area Network),and downloads content to be displayed as virtual information.

Note that the fifth sending/receiving device 112 of the HMD 100 b or theseventh sending/receiving device 405 of the mobile information terminal400 may be a wired sending/receiving device.

MODIFIED EXAMPLE 3

FIG. 12 is a schematic diagram showing an information processing system4 according to the modified example 3.

The hardware configuration of the information processing system 4 ofthis modified example is similar to the hardware configuration (FIG. 2)of the information processing system 1 of the first embodiment.

In the above embodiment, the CPU in the HMD executes the main process.To the contrary, in the information processing system 4 of this modifiedexample, a mobile information terminal as the input device 200 executesthe main process. For example, a smartphone or a handheld game machinemay be employed as the mobile information terminal.

The first sending/receiving device 108 of the HMD 100, for example,sends an image input signal of a real image obtained by the image-takingunit 101 to the input device (mobile information terminal) 200.

The third sending/receiving device 208 of the input device (mobileinformation terminal) 200, for example, sends an image output signal ofvirtual information to be displayed by the display unit 102 of the HMD100, to the HMD 100.

The CPU 212 of the input device (mobile information terminal) 200executes various processes in accordance with programs stored in thememory 205 as the function units described in the above embodiment.

Note that, in a case where the display unit 102 and the image-takingunit 101 are mounted in one apparatus (HMD 100), a superimpositionparameter is corrected (Step S110) based on positional relation betweenthe HMD 100 and the input device 200.

MODIFIED EXAMPLE 4

FIG. 13 is a schematic diagram showing an information processing system5 according to the modified example 4. FIG. 14 is a block diagramshowing the hardware configuration of the information processing system5 according to the modified example 4.

According to the above embodiment, a marker is projected and anoperation to virtual information is input by using the input device 200independent of the HMD 100. To the contrary, according to this modifiedexample, an input device is not provided separately. Only an HMD 100 cexecutes all the behaviors including projection of a marker and input ofan operation to virtual information.

The hardware configuration of the HMD 100 c is similar to the hardwareconfiguration of the HMD 100 of the first embodiment except that the HMD100 c does not include the first sending/receiving device 108, andfurther includes a modulation unit 113 connected to the CPU 103 and aprojection unit 114 connected to the modulation unit 113. The modulationunit 113 has a function similar to the function of the modulation unit209 of the input device 200 of the above embodiment. The projection unit114 has a function similar to the function of the projection unit 210 ofthe input device 200 of the above embodiment.

A user wears the HMD 100 c and moves his head to thereby control virtualinformation. For example, a user moves his head up/down/right/left tothereby move the projection position of a marker. As a result, it ispossible to move (drag) the display position of the displayed virtualinformation.

THE MODIFIED EXAMPLE 5

FIG. 15 is a schematic diagram showing an information processing system6 according to the modified example 5.

The information processing system 6 includes a plurality of HMDs 100A,100B, 100C, a plurality of input devices 200, and a server apparatus500. The server apparatus 500 holds content data of content to bedisplayed as virtual information.

The projection HMD 100A searches the server apparatus 500 for contentdata of content to be displayed as virtual information via a wireless orwired LAN (Local Area Network), and obtains the content data. Theprojection HMD 100A supplies content data obtained from the serverapparatus 500 to the non-projection HMDs 100B, 100C by using aclose-range wireless sending/receiving device (fourth sending/receivingdevice 111). As a result, the projection HMD 100A and the non-projectionHMDs 100B, 100C are capable of displaying the same content as virtualinformation.

MODIFIED EXAMPLE 6

FIG. 16 is a schematic diagram showing an information processing system7 according to the modified example 6.

The information processing system 7 includes a main processor apparatus600, a projection/image-taking device 700, and at least one displaydevice 800.

The main processor apparatus 600 instructs the projection/image-takingdevice 700 about the shape of a marker M to be projected, the locationof the marker M in real scenery S, and the like via close-range wirelessor wired communication. Further, the main processor apparatus 600obtains real image data from the projection/image-taking device 700. Themain processor apparatus 600 calculates object image data of virtualinformation based on the obtained real image data. The main processorapparatus 600 superimposes the calculated object image data on realimage data to thereby generate display data. The main processorapparatus 600 supplies the generated display data to the plurality ofdisplay devices 800 via wireless communication.

The projection/image-taking device 700 projects the marker M on the realscenery S. A stationary camera of the projection/image-taking device 700takes an image of the real scenery S to thereby obtain real image data.The projection/image-taking device 700 supplies the obtained real imagedata to the main processor apparatus 600.

The display device 800 displays display data obtained from the mainprocessor apparatus 600. The display device 800 is, for example, an HUD(Head-Up Display). Specifically, a digital signage, a transparentdisplay that may be installed on a desk or on a dashboard of a car, adisplay of a mobile information terminal, or the like may be employed asan HUD.

MODIFIED EXAMPLE 7

A non-visible region (infrared, ultraviolet, etc.) laser may be used asa light source for projecting a marker. As a result, it is possible notto allow a user who does not wear an HMD to visually recognize a markerand virtual information. Meanwhile, a user who wears an HMD is capableof visually recognizing virtual information. Further, a display unit ofan HMD may be processed such that a non-visible region (infrared,ultraviolet, etc.) laser is visually recognizable. As a result, a userwho wears an HMD is capable of visually recognizing a marker.

Note that the present application may employ the followingconfigurations.

-   (1) An information processing apparatus, comprising:    -   an image-taking unit configured to take an image of real scenery        to thereby obtain real image;    -   a marker-detecting unit configured to extract a marker image        from the real image, the marker image being an image of marker        projection light, the marker projection light being projected by        a projection device to the real scenery in order to provide        spatial first information, the first information being necessary        to display virtual information such that the virtual information        is superimposed on the real scenery, second information being        added to the marker projection light; and    -   an extracting unit configured to extract the second information        added to the extracted marker image.-   (2) The information processing apparatus according to (1), wherein    -   a measurable characteristic is added to the marker projection        light as the second information, and    -   the extracting unit is configured        -   to measure the characteristic based on an image of the            marker projection light, and        -   to extract the second information.-   (3) The information processing apparatus according to (1) or (2),    wherein    -   the measurable characteristic of the marker projection light is        at least one of light intensity (amplitude), wavelength        (frequency), and blinking periods.-   (4) The information processing apparatus according to any one of (1)    to (3), further comprising    -   an image generating unit configured to generate, based on the        first information, an image of virtual information displayed        such that the virtual information is superimposed on the real        scenery.-   (5) The information processing apparatus according to any one of (1)    to (4), wherein    -   the second information is identification information uniquely        identifying the projection device,    -   the extracting unit is configured to determine if the extracted        identification information indicates that the projection device        projects a display target of virtual information, and    -   the image generating unit generates, if the extracting unit        determines that the extracted identification information        indicates that the projection device projects a display target        of virtual information, an image of the virtual information.-   (6) The information processing apparatus according to any one of (1)    to (5), further comprising    -   a filter capable of detecting a characteristic of the marker        projection light, wherein    -   the marker-detecting unit is configured to detect a marker image        from the filtered real image.-   (7) The information processing apparatus according to any one of (1)    to (6), further comprising    -   a display unit configured to display an image of the virtual        information such that the image of the virtual information is        superimposed on the real scenery.-   (8) The information processing apparatus according to any one of (1)    to (7), wherein    -   the second information is configuration changing information of        virtual information displayed such that the virtual information        is superimposed on the real scenery, and    -   the image generating unit is configured to generate an image of        the virtual information based on the configuration changing        information of virtual information extracted by the extracting        unit.-   (9) The information processing apparatus according to any one of (1)    to (8), further comprising    -   a filter capable of detecting a characteristic of the marker        projection light, wherein    -   the marker-detecting unit is configured to detect a marker image        from the filtered real image.-   (10) The information processing apparatus according to any one    of (1) to (9), further comprising    -   a display unit configured to display an image of the virtual        information such that the image of the virtual information is        superimposed on the real scenery.

It should be understood that various changes and modifications to thepresently preferred embodiments described herein will be apparent tothose skilled in the art. Such changes and modifications can be madewithout departing from the spirit and scope of the present subjectmatter and without diminishing its intended advantages. It is thereforeintended that such changes and modifications be covered by the appendedclaims.

The invention is claimed as follows:
 1. An information processingapparatus, comprising: a Central Processing Unit (CPU) configured to:control an image sensor to take an image of a real scenery to obtain areal image; detect a marker image from the obtained real image, themarker image being an image of marker projection light projected by aprojection device to the real scenery, wherein the marker image includesa first information that includes a spatial information associated witha virtual information displayed on a display device and the virtualinformation is superimposed on the real image, and wherein theprojection device adds a second information to the marker projectionlight; and extract the second information added to the marker projectionlight, wherein the second information comprises at least one ofintensity or wavelength of the marker image as configuration changinginformation to change a configuration of the virtual informationsuperimposed on the real image, and wherein the configuration of thevirtual information comprises a size of the virtual informationdisplayed on the display device.
 2. The information processing apparatusaccording to claim 1, wherein the at least one of the intensity orwavelength of the marker image is added to the marker projection lightas the second information, and the CPU is further configured to: measurethe at least one of the intensity or wavelength of the marker image; andextract the second information.
 3. The information processing apparatusaccording to claim 1, wherein the CPU is further configured to generate,based on the first information, an image of the virtual informationdisplayed such that the virtual information is superimposed on the realscenery.
 4. The information processing apparatus according to claim 1,wherein the second information comprises an identification informationto uniquely identify the projection device, the CPU is furtherconfigured to: determine whether the identification informationindicates that the projection device projects a display target of thevirtual information; and generate the image of the virtual informationbased on the determination that the identification information indicatesthat the projection device projects the display target of the virtualinformation.
 5. The information processing apparatus according to claim2, further comprising a filter configured to detect the at least one ofthe intensity or wavelength of the marker image.
 6. The informationprocessing apparatus according to claim 1, further comprising thedisplay device configured to display the image of the virtualinformation such that the image of the virtual information issuperimposed on the real scenery.
 7. The information processingapparatus according to claim 1, wherein the CPU is further configured togenerate the image of the virtual information based on the configurationchanging information of the virtual information.
 8. The informationprocessing apparatus according to claim 7, further comprising a filterconfigured to detect a characteristic of the marker projection light. 9.The information processing apparatus according to claim 8, furthercomprising the display device configured to display the image of thevirtual information such that the image of the virtual information issuperimposed on the real scenery.
 10. The information processingapparatus according to claim 1, wherein the CPU is further configured tosend the second information to the projection device, the secondinformation being added to the marker projection light, wherein thesecond information is obtained by modulation of the second informationinto the marker projection light, the second information being receivedby the projection device from the information processing apparatus, andwherein the CPU is further configured to demodulate the secondinformation added to the marker projection light.
 11. The informationprocessing apparatus according to claim 1, wherein the secondinformation comprises location information specifying a location,wherein object data to be displayed as the virtual information is storedin the location, and the CPU is further configured to obtain, based onthe location information, the object data to be displayed as the virtualinformation.
 12. The information processing apparatus according to claim11, further comprising a filter configured to detect a characteristic ofthe marker projection light.
 13. The information processing apparatusaccording to claim 12, further comprising the display device configuredto display the image of the virtual information such that the image ofthe virtual information is superimposed on the real scenery.
 14. Theinformation processing apparatus according to claim 1, wherein theconfiguration of the virtual information further comprises displayposition of the virtual information and an angle of the virtualinformation.
 15. The information processing apparatus according to claim1, wherein the second information comprises the intensity of the markerimage as the configuration changing information to change theconfiguration of the virtual information superimposed on the real image.16. The information processing apparatus according to claim 1, whereinthe second information comprises the wavelength of the marker image asthe configuration changing information to change the configuration ofthe virtual information superimposed on the real image.
 17. Aninformation processing method, comprising: taking, by an image sensor,an image of a real scenery to obtain a real image; detecting a markerimage from the obtained real image, the marker image being an image ofmarker projection light projected by a projection device to the realscenery, wherein the marker image includes a first information thatincludes a spatial information associated with a virtual informationdisplayed on a display device and the virtual information issuperimposed on the real image; adding, by the projection device, asecond information to the marker projection light; and extracting thesecond information added to the marker projection light, wherein thesecond information comprises at least one of intensity or wavelength ofthe marker image as configuration changing information to change aconfiguration of the virtual information superimposed on the real image,and wherein the configuration of the virtual information comprises asize of the virtual information displayed on the display device.
 18. Aninformation processing system, comprising: a projection deviceconfigured to project a marker to a real scenery; and an informationprocessing apparatus, including: a Central Processing Unit (CPU)configured to: control an image sensor to take an image of the realscenery to obtain a real image; detect a marker image from the obtainedreal image, the marker image being an image of the marker projected bythe projection device to the real scenery, wherein the marker imageincludes a first information that includes a spatial informationassociated with a virtual information displayed on a display device andthe virtual information is superimposed on the real image, wherein theprojection device adds a second information to the marker projectionlight; and extract the second information added to the marker projectionlight, wherein the second information comprises at least one ofintensity or wavelength of the marker image as configuration changinginformation to change a configuration of the virtual informationsuperimposed on the real image, and wherein the configuration of thevirtual information comprises a size of the virtual informationdisplayed on the display device.
 19. The information processing systemaccording to claim 18, wherein the CPU is further configured to send thesecond information to the projection device, the second informationbeing added to the marker projection light, wherein the projectiondevice: receives the second information from the information processingapparatus; and modulates the second information received into the markerprojection light, and wherein the CPU is further configured todemodulate the second information added to the marker projection light.